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ABSTRACT 

A promising recent development in tertiary education involves 
the application of problem-based learning as a curricular vehicle to develop 
student talent. Problem-based learning (PBL) is common in professional 
education, such as in medical, law, and business schools, and is becoming 
increasingly common in pre-college education. However, it is less common in 
information systems education. The authors have successfully applied PBL to 
information systems courses, and this paper reports the results. There is a 
major difficulty in applying PBL as there is no formal methodology of 
assessing students' work. This paper explores this issue by creating an 
instrument for assessment in a PBL setting. The instrument has been 
specifically designed to evaluate the generic abilities and skills of 
information systems graduates. It is recommended that the instrument or a 
modified version of the instrument be used for assessing other subject areas 
when the PBL is applied. The instrument has been proved to be a successful 
tool for assessment of the PBL. The result of applying this assessment 
instrument to information systems courses is also presented. An appendix 
presents an outline of course objectives. Includes one table. (Contains 18 
references.) (Author) 
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ABSTRACT 

A promising recent development in tertiary education involves the application of problem-based learning as a 
curricular vehicle to develop student talent. Problem-based learning (PBL) is common in professional education, such 
as in medical, law, and business schools, and is becoming increasingly common in pre-college education. However, 
it is less comm on in information systems education. We have successfully applied PBL to information systems courses, 
and are reporting our results in this paper. Moreover, there is am ajor difficulty in applying P BL as there is no formal 
methodology of assessing students ' work. This paper explores this issue by creating an instrument for assessm ent in 
a PBL setting. The instrument has been specifically designed to evaluate the generic abilities and skills ofinformation 
systems graduates. We also recommend that the instrument or a modified version of the instrument be used for 
assessing other subject areas when the PBL is applied. The instrument has been proved to be a successful tool for 
assessment of the PBL. The result of applying this assessment instrument to our information systems courses will also 
be presented. 



INTRODUCTION 

Problem-based learning (PBL) isan educational strategy 
that uses problems as the starting point for student 
learning (Bligh, 1995). It is a curriculum design and 
teaching/leaming strategy, which recognizes the need to 
develop problem-solving skills as well as the necessity 
of helping students to acquire the necessary knowledge 
and skills (Boud & Feletti, 1997; Biggs 1999). The main 
issue is to reduce direct instruction as students assume 
greaterresponsibility fortheir own learning. Students are 
given ill-structured problems through which they 
develop high-orderthinking and problem-solving skills. 
The shift in the teaching and learning process is more 
student-centered than teacher-centered. The role of the 
teacher is to encourage student participation, provide 
guidance to students, offer timely feedback, and assume 
the role of learner as well (Aspy et al., 1993). Evidence 



of the success of problem-based learning as an 
instructional strategy is strongly positive, particularly in 
fostering increased knowledge retention (Norman & 
Schmidt, 1992), encouraging general problem-solving 
skills, for deep-biased students (Norman & Schmidt, 
1992; Lai, Tiwari & Tse, 1 997), promoting self-directed 
learning skills, and increasing intrinsic interest in the 
subject matter. Since its introduction as an instructional 
method used in the medical school at McMaster 
University in Ontario in the 1960s, PBL has spread to 
numerous educational institutions around the world. To 
the best of our knowledge, the PBL strategy has not 
become so popular in the field of computer science and 
information systems. We have applied the methodology 
for the first time in our respective institutions quite 
successfully. The course that we applied it in our 
discipline was Systems Analysis. Students had to take 
greater responsibility for coming up with a solution for 
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a given problem case. The details of this application 
follow this section. 

Aligning PBL activities and subsequent student 
assessment often proves to be difficult for teachers, with 
many PBL activities followed by traditional assessment 
confusing students by disrupting their understanding of 
teacher expectations. Traditional assessment generally 
requires that students memorize the content of teaching 
materials without understanding. Once the examination 
is over, everything has been forgotten. Students learn to 
pass examinations without understanding what they 
should have learnt. On the other hand, if assessment 
requires students to solve problems, then they collectas 
many sample solutions of problems as they can. The key 
issue is to assess students so that they learn what we 
want them to learn. Quite a few PB L assessm ent mod els 
have been adopted, but they may not be appropriate to 
assess the generic quality of IS graduates. In the 
remainder of this paper, we describe some of the most 
recentPBL assessment strategies that have been adopted 
by different scholars, present an instrument which is 
particularly designed for the assessment of IS subjects 
when the PBL approach to teaching is applied, and 
finally our experience of applying this assessment 
instrument to IS courses in the fall of 1 999. The main 
focus of this paper was the development and 
implementation of an assessment instrument when the 
PBL strategy is applied. 

PBL ASSESSMENT MODELS 

The study of problem-based learning assessment has 
been taken in two different directions by researchers, 
namely assessing the value of a problem-based learning 
curriculum, and student assessment in a problem-based 
learning approach. In this work we are concerned with 
the latter issue that is how to assess students’ work. 
When we first applied PBL to our information systems 
courses, the first problem that we encountered was how 
to assess students’ work in a PBL setting. Many 
instructors and researchers who apply or study the 
assessment of PBL have concerns about these issues. For 
example, Woods (1996) asks, “how do we handle tests 
in the context of small group, self-directed, self- 
assessment PBL?” There is, however, no general and 
systematic answer to this question, because students may 
be working alone, or in teams, and doing work that 
doesn’t fit into readily assessable products. Researchers 
have been working on PBL assessment for many years 
and have taken different approaches. These diverse 
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approaches in PBL assessment as pointed out by 
Sundberg (1999) are due to different factors that will 
impact on the success of student-active innovation. 
These include the culture of the institution, the 
department, or even a particular subject. Reis and 
Renzilli (1991) claim that the major weakness of 
historical and contemporary PBL efforts is the lack of 
formal student evaluation. In the literature, there are 
many studies of PBL assessment in different subject 
areas. Generally speaking, they all share the same 
concern. However, the approaches responding to these 
concerns differ from subject to subject and from 
institution to institution. In the following we present 
some of the most recent research results. 

Nowak and Plucker (1999) provide suggestions for 
aligning instructional activities and assessment in the 
PBL process. This suggestion assumes three categories 
of students. In the first option, the assumption is that 
students are professional in the subject area of the 
instruction, and the instructor is their supervisor. The 
second suggestion is that the PBL assessment should be 
structured in the area of the student outcome. They 
provide reasonable guidelines regardingthe instructor’s 
expectation from the students. Finally, the third 
suggestion states that instructors should hold off an 
assessment until the end of the activity or until the unit 
of work is complete. 

In the mental measu remen ts yearbook, Buros (1999) has 
compiled several instruments that can be used for PBL 
assessm ent. Many faculty of McM aster University have 
adopted a modified version of these instruments. Some 
of these instruments include testing of subject 
knowledge, problem-solving skills, metacognitive skills, 
lifetime learning skills, and critical thinking skills. 

Hicks (1998) classifies assessment of a PBL subject into 
four categories, namely teacher observation, student- 
produced written material and products, peer and self- 
evaluation, and feedback from the outside community. 
All of these four different classifications have the same 
outcome objectives, such as basic skills, and critical 
thinking skills. Hicks points out thatthe hardest to assess 
are the attributes and disposition of students, such as 
empowerment, diligence, empathy and pride. 

Woods (1998) has proposed several units of skills for 
assessmentof chemical engineeringcourses atMcM aster 
University. These units focus on individuals solving 
relatively well-defined problems, interpersonalskills and 
group problem-solving, and messy problem-solving 
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skills. Woods (1996) has also provided a general 
guideline for PBL assessment directed mostly towards 
chemical engineering courses. 

The SOLO taxonomy (Boulton- Lewis, 1998) is based on 
the study of outcomes in a variety of academic content 
areas. It stands for Structure of the Observed Learning 
Outcome. It has five levels of taxonomy: p restructural, 
unistructural, multistructual, relational and extended 
abstract. It is intended to measure the students’ 
quantitative and qualitative increase in understanding of 
the subject. In quantitative terms, it measures how much 
students can memorize and recognize terminology and a 
disorganized set of items. It also measures the students’ 
performance in qualitative terms. That is how students 
relate, apply, generate, reflectand theorize of the subject 
taught. At this level, students are able to apply theory to 
practice for problem-solving. 

Assessment by portfolio (Gibbs, 199 8) requires the 
student to be acquainted with the course objectives and 
is asked to provide evidence that learning relevant to 
those objectives has been achieved. This requires the 
learner to recognize the nature and quality of his or her 
own learning. The student presents his or her best 
‘learning’ against the objectives. Students have to use 
their judgments in assessing their own work. There are 
two drawbacks. First, the students may claim something, 
which he or she did not do. Second, the portfolio may 
require that students work excessively and create work 
both for themselves and for the teachers. 

Triple jump (Feletti, 1997) uses a three-step exercise, 
with the student evaluated at each step: step one deals 
with the problem case; diagnosing, hypothesizing, 
checking with the database, use of information, 
reformulating. Step two tests them on independent study: 
knowledge gained, levelof understanding, evaluation of 
information gained. Step three is concerned with final 
problem formulation: synthesis of key concepts, 

application to problem, self-monitoring, response to 
feedback. 

Researchers at the University of Wisconsin (Stone, 
1996) have adopted a new PBL assessment thatthey call 
ability-based assessment, in which they have identified 
nine generic abilities that a medical graduate should pass 
to be admitted into residency training. These abilities 
include appraisal, analysis, assessing own and peer 
performance, self-directed learning, handling stress, 
completing tasks, communications, consideration of 
professionalethicsin decision-making, and interpersonal 



skills. Multiple-choice testing has been replaced by this 
activity-based assessment. 

(Trevitt C. and Pettigrove M., 1995) have been using a 
combination of assessment and self-assessment to a fire 
science and management course using five criteria, 
namely class tests, oral debrief exercises, district 
committee fire management plan proposal, take home 
assignments and presentation. The main goal of their 
work is to increase the emphasis on inducing students 
into theprocess of self-evaluation according to specified 
criteria. 

The above PBL assessment strategies have been 
designed for specific objectives and for specific 
disciplines. To the best of the authors’ knowledge, no 
PBL assessment tool has been developed for information 
systems courses. When we first applied PBL teaching 
strategy to our courses the assessm ent of students’ work 
was a major concern for both students and instructors. 
Students were concerned how their work would be 
graded and instructors were also concerned bee ause there 
were no formal test or quizzes. To make the application 
of PBL strategy to our courses more practical and 
successful we have developed an assessment instmment 
for information systems courses that we have applied to 
our courses. The details of how the instrument was 
developed and subsequently applied are given below. 

PBL ASSESSMENT INSTRUMENT 

Information systems graduates (specially systems 
analysts) must have certain skills and ability and be 
prepared for industry. There is little doubt that 
information systems graduates should be able to 

• work independently and in a group environment, 

• solve a problem logically and systematically, with 
little or no supervision, 

• understand how to acquire knowledge for problem- 
solving, 

• have critical thinking and problem-solving ability, 
and 

• communicates effectively (both orally and in 
writing) 

Students must be equipped with the academic knowledge 
and the practical skills for problem-solving. They must 
understand how the theory is applied to practice, and 
know the circumstances n ecessary for achieving the cost- 
effective solution. There are several issues that concern 
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a teacher: first, what the students need to know 
academically; second, the skills they need to have; third, 
how they apply their skills and academic knowledge to 
know effectively when and why it is appropriate to do 
these things? The third should be an alignment between 
the curriculum objectives and assessm ent. That is to say, 
how well students know thesubject, how well they have 
learnt, and at what level. 

PBL activities support the development of ‘preferred IS 
graduates’, which also leads to long-life learners. When 
the PBL approach is adopted, there is some concern 
about how students are to be assessed with a practical 
and manageable assessment mechanism. The assessment 
of students should be aligned w ith their generic abilities. 
The assessment techniques mentioned above have 
targeted specific subject areas that are not necessarily 
suitable for IS courses. Thus, based on the previous work 
done by other researchers, we have created an 
assessment instrument specifically for IS courses. The 
instrument consists of a list of nine criterion skills, listed 
in Table 1. For each criterion skill, we have proposed a 
set of tasks that must be fulfilled to develop the given 
criterion skill. These tasks are listed immediately 
following Table 1. An instructor can use some or all of 
these tasks to assess students’ work for that skill. The 
authors used this instrument in theirinformation systems 
courses in the fall of 1999. The outcome of this 
experiment follows the list of criterion skills. Although 
the focus was on IS courses, most of these criteriacan be 
used fo r other subject a reas as well. 

Table 1 shows the criteriathat have been developed and 
used by the authors to assess students’ work in 
information systems courses. For each criterion skill, we 
have developed a set of tasks that can be used for 



assessment. The details of these tasks are listed in 
Appendix A . 

APPLICATION OF PBL AND 

ASSESSMENT INSTRUMENT 

The authors have taught systems analysis and design for 
a number of years using the traditional approach. This 
subject is taught at the upper level of a computer science 
(CS) and IS program. In an effort to develop students’ 
talents andenhance their learning, the authors decidedto 
apply the PBL strategy to the teaching of this course in 
their respective institutions for the first time in the fall of 
1999. The application of PBL was done with close 
coordination and communication between the authors. 

The systems analysis course in which we chose to apply 
PBL is offered once ayear. The course meets three times 
a week, two of which are lectures and the third meeting 
is lab/tutorial. The course was designed to be project- 
oriented, that is there was no formal test or quizzes, but 
all course requirements were fulfilled through projects. 
Every week there were two hours of lecture and one hour 
of lab/tutorial. Students worked in groups of up to six. 
They were assigned severalill-structuredproblem cases 
during the semester. For each problem case, students 
worked together to come up with a solution. During the 
lab/tutorial hour, the students worked on a certain aspect 
of their project that required group meeting, such as 
dividing the work among them selves, and setting future 
meeting dates. The instructor took the role of a coach 
monitoring their work. 

Students were assigned several ill-structured problem 
cases. For each problem case, they were required to 
submit both individual and group work in the form of 



TABLE 1 

LIST OF CRITERION SKILLS FOR PBL ASSESSMENT 



No 


Criterion S kills 


Description 


1 


Problem understanding 


Evidence of problem understanding and independent study 


2 


Use of resources 


Evidence of determination & utilization of information resources 


3 


Teamwork 


Evaluation of student’s teamwork success 


4 


Critical thinking 


Evidence of practical and optimal solution 


5 


Management skills 


Setting deadlines; finishing on time 


6 


Writing sk ills 


How well and professionally the work iswritten 


7 


Oral presentation 


How presentation is done 


8 


Team communication 


How well the student communicates with team members 


9 


Self-assessment 


Seriousness and accuracy of self-assessment 
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log their projects solutions as well as their projects. The 
assessment instrument have been used to grade students’ 
work. Each question in our assessment instrument were 
given a weight from 5 to 1 with 5 being the highest score 
and 1 being the lowest. This grading scale was used to 
grade their proje ct work . 

In the early stage of the PBL approach introduction, the 
change of students’ attitude towards the subject and 
especially towards the learning process became obvious. 
Students found the learning activities, such as teamwork, 
criticalthinking, and problem-solving more meaningful 
than the traditional approach of lecturing and taking 
multiple choice or short answer tests. Nevertheless, the 
students’ main concern was assessment. Questions like, 
“How will you grade our w ork?” were raised frequently. 
Application of assessment instrument reduced student’s 
anxiety significantly. This task could only be taken care 
of so quickly through the coordination and 
communication of the authors. After their first project, 
they developed more faith in our assessment 
methodology and were convinced that the assessment 
was relatively fair and accurate. For their subsequent 
projects, they showed less anxiety about their grade and 
focused on their learning activities. Students spent more 
time studying and preparing their projects in the PBL 
approach than in the traditional approach. The major 
focus ofthis course was the development and application 
of assessment instruments in PBL. The learning outcome 
has been very positive, and the informal feedback of 
students was favorable. Here are some quotes from 
students in our courses “ I learned systems analysis 
topics without much memorizing”, “ I enjoyed working 
in a team”, “PBL requires hard work and is a new 
learning experience”, and “The course was challenging 
and fun”. It is planned to have a formal survey on the 
students’ perception of our adopted assessment 
instruction, the result of which will be presented in 
another paper. 

CONCLUSION 

We have successfully applied the PBL strategy to IS 
courses and it was an exciting experience for both 
students and instructors. It proved to be superior to the 
traditional approach to teaching in many ways. First, 
students’ learning activities, including reading, 
searching, and writing, were much higher than when 
traditional teaching was applied to the same course. 
Second, the level of interaction among students was 
much higher. Third, the students had to keep a portfolio 
of their learning activities, which assisted them to be in 



control of the topics to be learnt. The major concern 
among educators who apply the PBL strategy is 
assessment of students’ work. Evaluation of student 
achievementis an important aspect of education, and the 
skills required for solving real-world problems must be 
included in that assessment. In addition, alignment of 
instruction and assessment is essential when the PBL 
approach is to be used. In response to this important 
concern, we have developed and applied an assessment 
instrument that is targeted towards the generic skills of 
IS graduates. The assessment instrument has created a 
unified framework for the instructors to evaluate 
students’ work; otherwise the assessment would have 
been very difficult. It has proved to be successful in 
many ways. It has significantly reduced students’ anxiety 
with regard to the accuracy and methodology of 
assessment as they realize the need to work towards the 
generic abilities of IS professionals. The instructors had 
a better ground for evaluation with a checklist that is 
easy to use for those who wish to apply the PBL strategy 
in their courses. Although IS graduate skills are the 
target of our instruments, it can also be fully or partially 
used for assessment of other subject areas. We have 
provided a useful tool for instructors who plan to apply 
PBL in their courses 
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1 . Problem Understanding Objective ; 

Students should be able to: 

• Describe all the aspects of the problem 

• Have a good idea of what needs to be done to solve the problem 

• Have created strategies to solve the problem 

• Have demonstrated the ability to work with limited supervision 

2. Use of Resources Objective : 

Students should be able to: 

• Prepare a printed copy of the needed material for his/her reading 

• Demonstrate his/her attempt to create other resources 

• Provide an accurate summary of the search work 

• Demonstrate evidence of electronic search 

• Relate the prepared information to the given problem 

• Evaluate the accuracy and credibility of the searched information 

• Create a relation between the prepared information and the real world problem 

3. Teamwork Objective : 

Students should be able to: 

• Demonstrate effective communication 

• Contribute at least his/her share of the work 

• Cope w ith conflicts 

• Evaluate team members’ work fairly and honestly 

• Share information with team members 

• Demonstrate leadership ability 

• Initiate discussion 

• Help other team members when they are in difficulty 

4. Critical Thinking Objective : 



Students should be able to: 

• Propose solutions to the given problem 

• Test their proposed solution 

• Propose alternate solutions 

• Provide reasons or justify the solutions 

• Present some background that support the solution 

• Evaluate and commend his/her solutions and the work of others 

• Present ideas/comments that are useful and constructive 

5. Project Management Objectives : 

Students should: 

• Have a good plan, schedule, and means of controlling the project 

• Be able to arrange tasks logically and systematically 

• Estimate resources reasonably 

• Submit their work on time 

• Provide a log of all meetings and communications with other team members 
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• Provide documented evidence of good communication with team members and outside organizations 

• Present a record of their submitted work 

6. Writing Objective : 

Students should: 

• Produce a professional report 

• Provide a good introduction and summary to their report 

• Perform a good problem analysis 

• Produce a report with good content 

• Provide alternative solutions with justification 

• Draw a reasonable conclusion 

• Provide references in their report 

• Demonstrate the ability to convert information into a meaningful report 

• Demonstrate any potential prospect for being successful systems analysts 

7. Oral Presentation Objective : 

Students m ust: 

• Explain their work clearly 

• Present the essential points of their work clearly 

• Use effective visual aids to support the presentation 

• Be able to answer questions during presentation 

• Be able to behave professionally during presentation 

• Be able to paraphrase their understanding of the topic during presentation 

• Be able to transfer their knowledge to the students in an appropriate way 

8. Team Communication Objective : 

Students m ust: 

• Have a good understanding of other team members’ suggestions and points 

• Demonstrate evidence of useful and constructive discussion 

• Make suggestions that provide guidelines to the team 

• Support their points through discussion 

• Provide evidence of making constructive suggestions 

• Provide concise and short messages and yet be able to resolve problems 

9. Self-Assessment Objective : 

Students should provide: 

• A list of what he/she contributed 

• A list of important topics learnt 

• A list of accurate assessment of other team members 

• A list of what he/she did not learn 

• A list of the meetings he/she did not take part in 

• A list of items that other students have done on his/her behalf 



36 



Proceedings of the 15 th Annual Conference of the International Academy for Information Management 

o BEST COPY AVAILABLE 




U.S. Department of Education 
Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 



Reproduction Basis 




This document is covered by a signed "Reproduction Release (Blanket)" 
form (on file within the ERIC system), encompassing all or classes of 
documents from its source organization and, therefore, does not require a 
"Specific Document" Release form. 



This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may be 
reproduced by ERIC without a signed Reproduction Release form (either 
"Specific Document" or "Blanket"). 




EFF-089 (1/2003) 



